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A  software  system  called  the  Fusion  Exploitation 
Tools  (FET)  underlies  an  automated  process 
for  alerting  intelligence  analysts  when 
abnormal  activity  occurs  in  an  area  under  their 
surveillance.  The  FET  consists  of  algorithms 
that  analyze  vehicle  and  ship  traffic.  It  works  in 
conjunction  with  the  Google  Earth  geospatial 
visualization  system,  presenting  the  user  with 
a  distributed  network-centric  way  of  visualizing 
this  information.  The  FET  is  unique  in  that 
it  automatically  analyzes  live  intelligence 
data  streams  in  near  real  time.  The  FET  also 
incorporates  direct  machine-to-machine 
requests  for  additional  data  collection,  providing 
both  enhanced  situation  awareness  and  a  timely 
and  effective  means  to  control  sensor  platforms. 


Intelligence  exploitation  is  the  process 

by  which  raw  intelligence  data  are  analyzed 
and  the  meaning  behind  the  data  is  deter¬ 
mined.  Historically,  intelligence  data  have 
been  imagery  (either  electro-optical,  infrared,  or  radar), 
and  exploitation  was  manually  performed  by  analysts. 
These  analysts  would  interpret  and  report  on  impor¬ 
tant  aspects  of  the  imagery,  such  as  a  submarine  leaving 
a  Soviet  naval  base  or  a  collection  of  tanks  readying  for 
a  field  exercise.  In  recent  years,  imagery  has  been  aug¬ 
mented  with  a  different  form  of  intelligence  data,  called 
moving-target  indication  (MTI).  MTI  provides  the  posi¬ 
tion  and  velocity  of  vehicles  and  ships  that  are  moving. 
One  of  the  challenges  with  MTI  data  is  the  extremely  large 
volume  of  information;  thus,  the  manual  exploitation  of 
MTI  is  highly  labor  intensive  and  time  consuming. 

The  Fusion  Exploitation  Tools  (FET),  developed  at 
Lincoln  Laboratory  under  the  sponsorship  of  the  National 
Geospatial-Intelligence  Agency  (NGA),  is  a  network¬ 
centric  system  that  performs  automated  exploitation 
in  near  real  time.  The  goal  of  the  FET  is  to  ingest  large 
amounts  of  intelligence,  surveillance,  and  reconnais¬ 
sance  (ISR)  data  and  automatically  cue  analysts  (or  sen¬ 
sors)  when  anomalous  activity  is  detected.  Consider  the 
example  of  the  screenshot  in  Figure  1.  It’s  a  Google  Earth 
view  generated  by  the  FET,  and  it  shows  several  regions 
under  surveillance  where  automated  MTI  analysis  indi¬ 
cates  normal  activity  (green)  and  several  regions  where 
abnormal  activity  was  detected  (red).  In  this  example,  the 
presence  of  several  convoys  and  fast-moving  vehicles  that 
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FIGURE  1.  A  Google  Earth  view  shows  situational  awareness  via  the  Fusion  Exploitation  Tools  (FET)  watchboxes.  Blue 
areas  are  regions  under  surveillance  for  which  no  intelligence,  surveillance,  and  reconnaissance  (ISR)  data  are  yet  available. 
Green  areas  indicate  that  normal  activity  was  detected,  while  red  areas  indicate  warnings  that  activity  is  outside  the  scope  of 
previously  observed  normal  behavior  for  that  area.  The  panel  at  left  provides  a  color-coded  historical  view  of  the  state  of  each 
watchbox  over  time. 


were  outside  the  bounds  of  normal  traffic  patterns  was  an 
anomaly  that  triggered  alerts. 

The  ability  of  the  United  States  and  its  coalition 
forces  to  develop  and  deploy  sensors  has  dramatically 
increased  the  need  for  systems  that  can  help  operators 
quickly  extract  knowledge  and  understanding  from  the 
ever-increasing  volumes  of  sensor  data  produced.  In  Jan¬ 
uary  2007,  NGA  and  the  Department  of  Defense’s  Space 
Radar  integrated  program  office  approached  Lincoln 
Laboratory  and  asked  for  concepts  on  how  a  network¬ 
centric  automated  exploitation  system  could  assist  with 
these  challenges.  At  that  time,  the  NGA  had  begun  explor¬ 
ing  Google  Earth  as  a  common  framework  within  which 
to  display  ISR  data  and  had  recently  deployed  Google 
Earth  servers  on  both  the  Secret  Internet  Protocol  Router 
Network  (SIPRNet)  and  the  Joint  Worldwide  Intelligence 
Communications  System.  Thus  the  development  of  the 
FET  began,  with  an  initial  emphasis  on  automated  exploi¬ 
tation  of  MTI  data,  an  architecture  designed  for  a  growth 


path  supporting  ingest  of  additional  ISR  data  types,  and 
a  design  goal  for  a  display  system  based  on  Google  Earth. 
FET  was  to  turn  Google  Earth  into  a  dynamically  updat¬ 
ing  situation-awareness  display  (Figure  1),  automatically 
generating  indications  and  warnings  in  near  real  time  as 
live  ISR  data  streams  were  received. 

An  example  of  how  the  FET  can  be  used  is  to  moni¬ 
tor  suspicious  areas  for  the  presence  of  a  convoy.  Such 
monitoring  is  typically  employed  when  an  attack  from  a 
vehicle-borne  improvised  explosive  device  has  begun.  By 
automatically  detecting  a  convoy,  the  FET  could  cue  the 
operator  that  this  group  of  vehicles  may  be  suspicious.  If 
the  operator  believes  that  this  situation  warrants  further 
investigation,  he  or  she  can  use  the  FET  Convoy  Alert  to 
automatically  request  an  unmanned  aerial  vehicle  (UAV) 
to  take  a  closer  look. 

Several  MTI  analysis  tools  already  exist.  The  most 
popular  of  these  are  the  Moving  Target  Exploitation  sys¬ 
tem  (MTIX,  and  PC  MTIX),  DotMatrix,  and  the  ISR 
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About  Google  Earth 


Google  Earth  is  an  interactive, 

three-dimensional  data  viewer  origi¬ 
nally  developed  by  Keyhole,  Inc. 
Centralized  servers  stream  the  vir¬ 
tual  globe,  comprising  underlying 
geographical  data,  to  the  Google 
Earth  client  software. 

Google  Earth  supports  the 
display  of  spatiotemporal  data  lay¬ 
ered  on  top  of  the  underlying  globe 
data,  known  as  the  Google  Earth 
“data  brick.”  The  version  of  the  data 
brick  that  Google  provides  on  the 
Internet  has  numerous  interesting 
built-in  layers  above  and  beyond 
the  basic  capabilities  of  satellite 
imagery  draped  over  digital  terrain 
elevation  data  (earth  surface  height 
information).  It  includes  such  basic 
geographic  information  as  roads, 
city  names,  and  state  and  country 
boundaries. 

One  powerful  feature  of 
Google  Earth  is  that  its  maps  can 
link  to  content  created  by  individual 
users,  such  as  Wikipedia  entries, 
digital  photos,  and  geospatially 


FIGURE  A.  This  Google  Earth  image  of  Niagara  Falls  is  an  example  of  the 
type  of  three-dimensional  imagery  available  as  a  backdrop  to  the  situational¬ 
awareness  layers  that  can  be  provided  by  the  FET. 


tagged  YouTube  video.  Toward 
this  end,  Google  Earth  provides 
for  an  entire  custom  layer  of  user- 
specified  data,  defined  via  the  Key¬ 
hole  Markup  Language  (KML),  an 
XML  schema  specific  to  Google 
Earth.  These  KML  files  allow  a  user 
to  specify  geometric  primitives 
(points,  lines,  polygons),  imagery 


overlays,  and  temporal  data  (the 
ability  to  play  back  time-sequence 
data).  They  also  provide  the  abil¬ 
ity  to  have  the  Google  Earth  client 
software  periodically  poll  a  server 
to  check  for  the  arrival  of  new  KML 
content.  This  function  is  achieved 
via  the  KML  NetworkLink  concept. 


Forensics  Tool.  Figure  2  and  Figure  3  show  displays  of 
data  from  the  Lincoln  Multimission  ISR  Testbed  (LiMIT) 
by  PC  MTIX  and  DotMatrix,  respectively.  All  of  these  are 
stand-alone  software  tools  that  allow  analysts  to  investi¬ 
gate  MTI  data  sets.  None  of  these  tools,  however,  were 
designed  to  perform  automated  near-real-time  exploita¬ 
tion  or  analysis.  In  fact,  the  FET  has  demonstrated  utility 
as  a  cueing  system  for  these  other  tools.  In  other  words, 
the  FET  can  be  used  as  a  first-order  filter  to  cue  an  ana¬ 
lyst  where  abnormal  activity  is  happening.  Once  cued,  the 
operator  then  uses  one  of  the  above  MTI  analysis  tools  to 
investigate  further. 


How  FET  Works 

The  FET  is  a  web  application  server  (or  portal)  that 
works  in  conjunction  with  the  capabilities  provided  by 
the  Google  Earth  client.  The  FET  does  not  need  to  be 
installed  on  individual  users’  computers.  Instead,  all  of  the 
features  of  the  FET  are  available  via  the  network  through 
a  standard  web  browser.  The  ability  to  add  or  delete  ISR 
data  from  the  system  is  restricted  to  users  with  adminis¬ 
trative  rights,  but  all  users  can  view  the  live  or  archived 
data,  or  customize  the  view  according  to  their  needs.  The 
software  is  written  primarily  in  Java  and  operates  within 
the  framework  of  the  Apache  Tomcat  system. 
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The  system  consists  of  three  large  areas  of 
functionality:  data  ingest,  automated  exploita¬ 
tion,  and  display  update.  Data  ingest  is  performed 
in  a  variety  of  ways,  all  of  which  use  standard  ISR 
data  formats. 

Data  Ingest.  In  the  first  data-ingest  approach, 
the  FET  monitors  a  directory  (or  shared  network 
folder)  for  the  arrival  of  new  data.  This  method 
is  used  when  a  sensor  generates  file-based  MTI 
reports.  Consider  an  example  in  which  data  are 
being  produced  by  LiMIT.  As  data  are  collected, 
new  MTI  files  are  generated  and  sent  wirelessly 
to  the  LiMIT  ground  station.  There  the  data  files 
are  stored  in  a  folder  that  is  constantly  monitored 
by  the  FET.  The  saved  files  are  automatically  read 
and  parsed,  and  their  contents  are  stored  in  a  rela¬ 
tional  database.  (The  current  implementation  of 
the  FET  uses  the  searchable  geospatial  database 
PostgreSQL.)  The  second  approach  is  to  listen  to 
the  network  for  ISR  data  broadcasts.  As  in  the 
file-based  approach,  data  received  through  the 


FIGURE  2.  The  PC  MTIX  Display  Client  (MDC)  is  a  common  tool 
used  for  performing  moving-target  indication  (MTI)  analysis.  The 
screenshot  shows  the  position  of  the  Lincoln  Multimission  ISR 
Testbed  (LiMIT)  aircraft  (yellow  aircraft  icon,  upper  right)  at  the  point 
in  time  it  was  collecting  a  single  MTI  radar  beam  dwell  (yellow  rectan¬ 
gle,  center)  during  the  Empire  Challenge  2007  field  exercise. 


File  Tools  Help 


FIGURE  3.  The  DotMatrix  tool  (shown  viewing  LiMIT  radar  MTI  data)  can  be  used  to  perform  detailed  analysis  of  indi¬ 
vidual  MTI  detections.  However,  with  the  large  volume  of  MTI  data  regularly  collected,  a  tool  like  the  FET  is  needed  to  help 
the  analyst  focus  on  what  subset  of  MTI  to  investigate. 
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network  are  processed  and  stored  in  the  database.  Both 
of  these  approaches  are  fully  automated— once  the  FET 
has  been  started  and  the  proper  parsers  have  been  loaded, 
no  user  interaction  is  required.  The  third  approach  for 
delivering  data  to  the  FET  is  to  use  a  standard  web-based 
upload  form  that  allows  the  user  to  explicitly  load  the 
desired  data  into  the  system. 

Having  three  different  ways  of  importing  data  gives 
the  FET  significant  flexibility,  and  it  can  operate  well  in 
just  about  any  network  architecture.  Moreover,  FET’s  abil¬ 
ity  to  ingest  data  in  all  three  manners  makes  it  compat¬ 
ible  with  a  wide  range  of  sensor  systems.  For  example,  the 
Joint  Surveillance  Target  Attack  Radar  System  (JSTARS) 
is  an  aircraft  that  generates  MTI  and  streams  it  over  the 
network;  it  does  not  generate  file-based  MTI.  LiMIT,  on 
the  other  hand,  is  file  based  and  does  not  stream  data 
over  the  network.  By  supporting  both  types  of  data  feed, 
FET  can  work  with  both  JSTARS  and  LiMIT  (as  well  as 
numerous  other  MTI  sources). 


Exploitation  Process.  The  exploitation  process  runs 
asynchronously  to  the  data  ingest,  allowing  the  analysis 
algorithms  to  execute  while  new  data  are  being  read  and 
stored  in  the  database.  The  exploitation  algorithms  use 
a  sliding  time  window  to  analyze  chunks  of  MTI  data— 
typically  15  seconds’  worth  at  a  time.  The  algorithms 
are  implemented  in  Matlab  and  operate  by  monitoring 
areas  of  interest,  called  watchboxes.  These  watchboxes 
are  defined  in  Google  Earth,  dropped  into  a  special  folder 
(or  upload  form)  on  the  FET,  and  stored  in  the  database. 
When  the  Matlab  exploitation  process  is  started,  it  loads 
the  watchboxes.  As  it  periodically  polls  the  database,  the 
system  is  on  the  lookout  for  the  appearance  of  new  ISR 
data  in  each  watchbox. 

In  Figure  1,  the  watchboxes  are  updated  as  data 
arrive.  In  this  example,  the  Region  82  watchbox— the 
red  area  near  the  center  of  the  screenshot— has  detected 
abnormal  activity  (fast  movers),  while  the  green  regions 
indicate  normal  activity  and  the  blue  areas  show  that  no 
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FIGURE  4.  When  a  target  is  highlighted,  textual  information  is  presented.  A  pop-up  window  shows  statistical  details  of  FET 
exploitation  algorithm  calculations  on  demand. 
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Cursor  on  Target 

Cursor  on  target  (CoT)  has  emerged  as  a 

standard  XML  messaging  format  for  sharing  situa¬ 
tion-awareness  information.  The  FET  currently  does 
not  exploit  CoT  data.  However,  there  does  exist  a 
CoT-to-KML  converter  that  allows  Google  Earth  to 
display  CoT  state  data  (e.g.,  position,  orientation, 
and  identification)  in  conjunction  with  FET  data. 

CoT  also  defines  a  message  format  for  requesting 
additional  ISR  data.  In  addition  to  supporting  the 
automated  machine-to-machine  CoT  task  request 
and  having  the  ability  to  generate  CoT  tracks,  the 
FET  also  displays  the  current  status  of  a  CoT  task 
(whether  it’s  been  acknowledged,  whether  the  sen¬ 
sor  has  agreed  to  collect  the  data,  and  whether  the 
data  collection  has  been  completed). 

current  information  is  available.  As  the  algorithms  pro¬ 
cess  new  data,  they  compare  the  current  information  with 
statistical  data  previously  calculated  for  that  watchbox 
and  determine  whether  the  new  data  are  within  normal 
statistical  limits. 

Because  average  vehicle  speed  is  one  component  of 
the  analysis,  different  regions  can  have  different  statisti¬ 
cal  limits.  A  vehicle  traveling  45  mph  may  be  considered 
normal  in  one  region,  but  in  another  it  may  trigger  an 
alert  that  the  vehicle  is  traveling  faster  than  average.  One 
way  that  FET  stands  alone  among  MTI  analysis  tools  is 
that  the  user  has  access  to  all  of  the  statistical  informa¬ 
tion  that  underlies  the  reasoning  behind  an  alert.  When 
an  alert  is  generated,  the  details  are  displayed  in  a  Google 
Earth  pop-up  message  (Figure  4).  The  pop-up  also  con¬ 
tains  a  hyperlink  that  leads  to  a  web  page  containing 
statistical  charts  (generated  on  demand)  detailing  the 
rationale  behind  the  alert.  The  results  of  the  exploitation 
algorithms  are  then  time  stamped  and  stored  back  into 
the  database,  and  the  results  are  converted  into  Keyhole 
Markup  Language  (KML)  for  display  in  Google  Earth. 

Display  Update.  Key  to  FET’s  value  is  its  ability  to 
dynamically  update  Google  Earth.  The  core  of  the  design 
revolves  around  the  KML  NetworkLink  construct.  Two 
broad  categories  of  information  utilize  this  approach  in 


the  FET— the  display  of  raw  ISR  data  as  they  arrive,  and 
the  dynamic  update  of  exploitation  results. 

For  raw  data,  especially  for  large-volume  data  such  as 
that  produced  by  MTI,  the  FET  needs  a  way  to  limit  the 
volume  of  data  loaded  into  Google  Earth  and  to  update 
the  data.  The  approach  that  we  chose  uses  the  concept  of  a 
regular  grid  overlaid  on  the  exercise  area,  as  shown  in  Fig¬ 
ure  5.  This  grid  is  itself  a  KML  file  and  is  loaded  from  the 
FET  web  portal  (new  grids  can  be  easily  generated  from 
the  portal  to  support  new  exercise  areas).  Each  cell  in  the 
grid  contains  a  NetworkLink  that  periodically  polls  the 
FET  for  new  data.  At  a  user-defined  interval  (the  fastest 
refresh  available  is  1  s),  Google  Earth  attempts  to  reload 
a  KML  file  specified  by  the  NetworkLink.  Upon  receiving 
this  request,  the  FET  regenerates  the  KML  content  for 
that  grid  cell  and  passes  the  information  back  to  Google 
Earth,  which  then  displays  the  new  results. 

In  the  grid-cell  construct,  the  visual  display  of  the 
grid  works  independently  of  the  data  update,  allowing  the 
user  to  disable  the  visual  grid  but  to  retain  the  function¬ 
ality  of  dynamic  data  updates.  This  structure  allows  the 
screen  to  be  less  cluttered  while  maintaining  the  dynamic 
update  capability.  The  other  useful  feature  of  the  data  grid 
is  the  ability  to  indicate  that  data  are  available  without 
requiring  the  user  to  see  all  of  the  underlying  atomic  data 
elements:  cells  for  which  data  are  available  are  shaded 
yellow.  This  capability  is  another  mechanism  to  reduce 
screen  clutter,  and  it  guarantees  that  the  user  is  presented 
raw  data  only  in  grid  cells  that  are  of  interest. 

The  dynamic  update  of  exploitation  results  works  in 
a  similar  fashion.  The  watchboxes  are  originally  created 


FIGURE  5.  Six  yellow  grid  cells  show  presence  of  MTI;  one 
cell  (2nd  row,  3rd  column)  is  enabled  to  show  raw  MTI  data. 
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FIGURE  6. The  FET  makes  the  process  of  performing  a  machine-to-machine  tip  and  cue  extremely  easy.  Operators  click  on 
the  alert  (the  four-vehicle  convoy  detection  shown  above)  and  manually  review  the  automated  warning  message  from  the  FET. 
If  they  determine  that  they  would  like  additional  sensor  coverage,  they  click  on  the  Request  ISR  Collection  link,  which  brings 
up  a  task  request  form  (with  the  fields  automatically  populated  by  the  FET).  The  final  step  is  to  select  the  sensor  platform 
from  a  drop-down  list  of  available  resources  and  click  Send  Task. 


in  Google  Earth,  then  loaded  into  the  FET,  where  they 
have  the  associated  NetworkLink  control  mechanisms 
added.  The  user  then  loads  these  watchboxes  via  the  FET 
web  portal,  and  the  NetworkLink  objects  poll  the  FET  in 
exactly  the  same  way  that  the  data  grid  cells  do.  The  visual 
result  is  watchboxes  that  change  color  as  new  exploitation 
results  are  calculated. 

When  the  FET  generates  an  indication  or  warning,  it 
generates  a  piece  of  KML  code  that  uses  the  Placemark 
object— a  clickable  Google  Earth  icon  that  the  user  can 
interrogate  for  detailed  information  on  the  exploitation. 
The  Placemark  provides  two  paths  for  dissemination  of 
the  exploitation  results. 

In  the  first  path,  the  exploitation  result  is  textual  data, 
along  the  lines  of  a  military  situation  report  that  contains 
the  essential  elements  of  information:  what’s  happening, 
where  it’s  happening,  and  who’s  reporting  the  informa¬ 
tion.  The  user  can  then  copy  and  paste  the  text  data  into 
e-mail  or  online  chat  windows. 

The  second  way  to  disseminate  the  exploitation  infor¬ 
mation  via  a  Placemark  connects  the  automated  exploita¬ 
tion  process  with  a  semiautomated  collection  request.  In 
this  path,  the  FET  exploitation  result  contains  a  hyperlink 


to  a  cursor-on-target  (CoT)  task  request  form  using  the 
standard  CoT  message  format  (see  the  sidebar,  “Cursor 
on  Target”  on  page  71)*  Clicking  this  hyperlink  populates 
all  of  the  essential  information  on  the  tasking  request,  as 
shown  in  Figure  6.  This  approach  allows  the  user  to  easily 
perform  a  machine-to-machine  tasking  request,  providing 
a  mechanism  to  cue  narrow-field-of-view  sensors  (such  as 
a  UAV)  from  wide-held- of- view  data  such  as  MTI. 

A  recent  addition  to  the  FET  provides  yet  a  third 
mechanism  to  disseminate  results.  This  manual  process 
involves  the  generation  of  a  track— a  series  of  spatiotempo- 
ral  reports  associated  with  a  specific  vehicle— that  is  then 
sent  onto  the  network  via  CoT.  When  FET  users  see  indi¬ 
cations  that  a  suspicious  vehicle  is  moving  in  an  area  of 
interest,  they  can  pull  up  a  web  form  and  fill  out  the  fields 
(latitude,  longitude,  track  ID,  and  comments)  and  send 
the  CoT  track  out  onto  the  network  (FET  automatically 
time-stamps  the  data).  Contact  information  that  is  sent  out 
via  a  CoT  track  will  appear  in  any  application  that  ingests 
CoT  data,  including  FalconView— a  geographical-data- 
display  software  package  used  widely  by  the  Department 
of  Defense.  This  method  allows  FET  results  to  be  sent  to 
users  who  don’t  have  access  to  the  Google  Earth  viewer. 
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Moving-Target  Indication 

MTI  reports  are  an  ISR  data  product  provided  by  numer¬ 
ous  radar  sensors  (including  Lincoln  Laboratory’s  own 
LiMIT  radar,  which  flies  aboard  a  modified  Boeing  707). 
MTI  data  are  normally  disseminated  in  a  standard  format 
(known  as  STANAG  4607)  and  viewed  through  one  of 
the  MTI  analysis  tools  previously  described.  In  addition 
to  being  able  to  display  raw  MTI,  the  FET  runs  several 
additional  algorithms  on  the  data. 

The  first  algorithm  takes  the  position  of  the  raw  MTI 
detection  (the  coordinate  of  the  MTI  detection  is  generally 
referred  to  as  the  MTI  dot)  and  projects  it  onto  the  nearest 
road.  MTI  generally  has  very  small  measurement  errors 
along  the  line  of  sight  to  the  sensor.  However,  cross-range 
errors— that  is,  perpendicular  to  the  sensor— can  be  on  the 
order  of  a  football  field  or  more.  Such  imprecision  makes 
it  difficult  to  interpret  a  display  of  MTI  dots.  The  FET 
road  projection  starts  at  the  position  of  the  MTI  dot  and 
begins  projecting  in  both  directions  along  the  cross-range 
error  segment  until  the  first  road  segment  is  encountered. 
This  will  be  the  position  of  the  projected  dot. 

Knowing  the  line-of-sight  velocity  (which  is  in  the 
original  MTI  data  report)  and  the  orientation  of  the  road 
permits  calculation  of  the  velocity  component  along  the 
road,  giving  both  a  direction  of  travel  on  the  road  and 
an  estimated  true  road  speed.  The  FET  display  indicates 
direction  of  travel  by  using  an  oriented  triangle.  In  exist¬ 
ing  MTI  analysis  tools,  the  color  of  the  MTI  dot  indicates 
movement  toward  the  sensor  (magenta)  or  away  from  it 
(yellow).  However,  in  a  multisensor  environment,  and 
even  in  an  environment  where  the  sensor  is  orbiting, 
this  color  scheme  becomes  confusing.  In  the  FET,  there¬ 
fore,  the  color  of  the  MTI  dot  (both  the  raw  dot  and  the 
projected  dot  triangle)  is  based  on  the  signal-to-noise 
ratio  (SNR)  of  the  detection.  High  SNR  values  (12  dB 
and  above)  indicate  strong  detections  and  are  likely  to 
be  vehicles,  which  get  color-coded  yellow.  Low  SNRs  (4 
dB  and  below)  are  possible  false  alarms,  color-coded  red. 
Values  in  between  are  shown  in  orange. 

Figure  7  shows  the  MTI  detections  projected  onto  a 
road  network.  The  triangles  show  direction  of  travel.  All 
the  detections  are  yellow,  indicating  high- SNR  returns— 
that  is,  likely  vehicles.  The  ends  of  the  white  lines  cor¬ 
respond  to  the  original  dot  location,  and  the  white  line 
indicates  the  projection  along  the  cross-range  error  to 
the  nearest  road.  Notice  also  the  orange  paddle  with  the 


FIGURE  7.  Projected  MTI  detections  provide  enhanced  sit¬ 
uational  awareness.  The  raw  position  of  the  MTI  detections 
are  at  the  ends  of  the  white  lines.  Without  the  FET,  it  would 
be  difficult  to  interpret  where  the  vehicles  really  were  and  in 
what  direction  they  were  heading.  The  FET  projects  these 
detections  onto  the  nearest  road  and  displays  a  triangle  rep¬ 
resenting  the  estimated  location  and  direction  of  travel  of 
each  vehicle.  The  color  of  the  triangles  is  based  on  the  signal 
strength  of  the  detections. 

number  3  in  it,  labeled  “Convoy”  to  indicate  the  detection 
of  a  three-vehicle  convoy  by  the  FET.  (A  convoy  is  defined 
as  two  or  more  vehicles  traveling  in  the  same  direction, 
with  approximately  the  same  speed  and  relatively  small 
inter-vehicle  spacing.) 

The  other  types  of  exploitation  the  FET  performs  on 
MTI  are  determination  of  fast  movers  and  slow  movers 
(vehicles  moving  at  speeds  higher  or  lower  than  the  aver¬ 
age);  higher  or  lower  level  of  activity,  relative  to  a  pre¬ 
defined  norm;  and  the  presence  of  an  abnormally  large 
convoy  or  of  an  abnormally  large  number  of  convoys. 
All  of  these  alerts  cause  the  watchbox  region  to  turn  red. 
In  addition,  the  fast-  and  slow-mover  alerts  and  convoy 
detections  display  an  alert  paddle  showing  their  location. 
While  the  FET  allows  the  user  to  display  raw  MTI,  the 
display  is  often  easier  to  understand  when  only  the  pro¬ 
jected  dots  and  exploitation  results  are  shown. 

Fusion 

In  the  ISR  community,  fusion  has  two  broad  definitions. 
The  first  is  the  ability  to  have  a  large  variety  of  infor¬ 
mation  available  for  display  in  distinct  layers  within  the 
same  application.  Such  layers  may  include  the  underlying 
earth  globe  and  maps,  road  networks,  air  traffic,  ground 
vehicle  activity,  tracks,  and  exploitation  results.  Users 
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FIGURE  8.  During  a  field  data  collection  from  Point  Dume  State  Beach  in  Malibu,  Calif.,  in  June  2008,  the  FET 
recorded  and  ingested  data  from  the  maritime  Automatic  Identification  System  (AIS).  One  of  the  ships  (the  Sanko 
Dynasty)  was  detected  as  a  fast  mover,  with  a  speed  faster  than  the  normal  speed  of  maritime  traffic  in  Long  Beach 
Harbor.  This  alert  caused  a  warning  state  to  be  activated,  indicated  by  the  Long  Beach  Harbor  watchbox  (shaded  red), 
alerting  the  user  to  abnormal  activity  in  that  area. 


can  turn  on  (or  off)  the  layers  that  they’re  interested  in. 
This  concept  is  referred  to  as  a  user-defined  operational 
picture  (UDOP).  Google  Earth,  with  its  ability  to  eas¬ 
ily  present  a  large  variety  of  layers  of  information  under 
user  control,  has  emerged  as  the  preferred  backbone  of 
a  UDOP  system. 

The  second  definition  of  fusion  refers  to  software 
algorithms  that  operate  on  various  types  of  ISR  data. 
There  are  several  ways  in  which  different  ISR  data  types 
are  fused  by  the  FET  to  get  a  more  coherent  understand¬ 
ing  of  the  surveillance  area.  The  sections  that  follow 
describe  two  additional  ISR  data  types— the  maritime 
Automatic  Identification  System  and  the  Constant  Hawk 
persistent  surveillance  system— and  how  the  FET  pro¬ 
cesses  their  data. 

Automatic  Identification  System 

An  emerging  standard  for  the  display  of  maritime  vessel 
information  is  the  Automatic  Identification  System,  or 
AIS.  AIS  is  a  set  of  radio-frequency  messages  that  indi¬ 


cate  the  identification  of  a  ship,  its  intended  port,  its  esti¬ 
mated  time  of  arrival,  and  its  present  position,  heading, 
and  velocity. 

The  FET  reads  AIS  messages  and  displays  them  in 
Google  Earth;  it  also  performs  exploitation  analysis  on 
the  data.  Maritime  watchboxes  are  defined,  and  AIS  activ¬ 
ity  in  the  area  is  analyzed.  The  algorithms  evaluate  higher 
than  normal  (and  lower  than  normal)  activity  and  per¬ 
form  the  same  type  of  fast-mover  and  slow-mover  detec¬ 
tion  as  for  MTI.  Another  function  generates  an  alert  if 
several  ships  appear  to  rendezvous.  Figure  8  shows  AIS 
data  collected  off  Point  Dume,  Calif.,  and  exploited  by  the 
FET,  with  a  fast-mover  alert  highlighted. 

An  additional  algorithm  takes  into  account  both  AIS 
and  MTI  data.  This  multi-intelligence  fusion  algorithm 
attempts  to  detect  false  AIS  position  reports.  Such  a  situ¬ 
ation  could  arise  if  a  nefarious  ship  doesn’t  want  its  true 
position  known  and  so  starts  sending  out  false  position 
information.  The  FET  uses  radar  MTI  information  to 
confirm  (or  reject)  the  presence  of  a  ship  in  the  vicinity  of 
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FIGURE  9.  The  results  of  the  Constant  Hawk  video  MTI  exploitation  are  enhanced  with  short 
snippets  of  video  (called  video  chips),  displayed  in  a  pop-up  window  in  this  notional  example. 


an  AIS  report.  The  FET  does  so  by  looking  for  high-SNR 
MTI  detections  in  the  vicinity  of  an  AIS  position  report.  If 
the  AIS  position  can  be  corroborated  with  the  MTI  data, 
then  the  FET  indicates  it  by  using  the  same  type  of  paddle 
icon  that  it  displays  for  convoys,  fast  movers,  and  other 
activity  of  concern. 

Constant  Hawk  Integration 

Constant  Hawk  is  a  wide-area,  large-format  sensor  used 
for  collecting  large  swaths  of  imagery  (roughly  12  by 
12  km),  updated  twice  per  second.  This  imagery  prod¬ 
uct  is  used  for  viewing  persistent  surveillance  video  in 
various  high-conflict  areas  of  the  world.  The  imagery  is 
of  high  enough  resolution  that  individual  vehicles  can 
be  detected  and  tracked.  The  FET  integrates  Constant 
Hawk  data  in  several  ways. 

The  Constant  Hawk  system  performs  image  process¬ 
ing  algorithms  on  the  data  it  collects  and  is  able  to  identify 
moving  vehicles  in  its  video.  The  FET  converts  these  Con¬ 
stant  Hawk  detections  into  the  STANAG  4607  format  so 
that  they  can  then  be  treated  in  the  same  fashion  as  any 
other  MTI.  This  standardization  allows  the  FET  to  per¬ 
form  the  same  automated  exploitation  of  the  video  MTI 
that  it  does  on  the  radar  MTI.  When  an  alert  is  detected, 
FET  generates,  on  the  fly,  a  short  video  snippet  (known 
as  a  video  chip)  in  the  direct  vicinity  of  the  alert,  with  a 
time  span  of  plus  or  minus  10  seconds  from  when  the 
alert  was  generated.  These  videos  have  the  same  Macro¬ 


media  Flash  format  (.swf) 
as  YouTube  videos.  Such 
compatibility  is  important 
because  it  allows  the  FET 
to  generate  pop-up  alert 
windows  in  Google  Earth 
that  contain  the  Constant 
Hawk  video  chip.  Because 
this  video  chip  is  a  Flash 
movie,  the  user  can  pan, 
zoom,  rewind,  and  replay, 
just  as  with  any  other 
Flash  movie  (Figure  9). 

The  other  way  that 
the  FET  has  integrated 
Constant  Hawk  data  is  by 
supporting  the  display  of 
the  high-resolution  Con¬ 
stant  Hawk  image  still  frames  in  Google  Earth  (Figure  10). 
Normally,  high-resolution  imagery  is  built  directly  into  the 
Google  Earth  ground  plane— a  time-consuming  process 
that  requires  the  Google  Earth  data  brick  to  be  updated  on 
a  centralized  server.  Alternatively,  the  data  can  be  down- 
sampled  and  a  lower-resolution  version  displayed.  The 
FET,  though,  has  incorporated  multiresolution  tile  pro¬ 
cessing  for  display  of  these  high-resolution  images.  Also 
known  as  image  pyramids,  or  mip-maps,  such  processing 
displays  low-resolution  versions  if  the  viewer’s  perspective 
is  far  away  from  the  image.  As  the  viewer  gets  closer  and 
closer,  higher-resolution  tiles  are  paged  in. 


FIGURE  10.  Constant  Hawk  imagery  displayed  in  Google 
Earth:  as  the  viewer  zooms  in,  higher-resolution  images  are 
paged  in  to  replace  lower-resolution  ones. 


VOLUME  18,  NUMBER  1,  2009  ■  LINCOLN  LABORATORY  JOURNAL  75 


AUTOMATED  INDICATIONS  AND  WARNINGS  FROM  LIVE  SURVEILLANCE  DATA 


FIGURE  11.  Scenes  from  the  Empire  Challenge.  A: 
exterior  of  the  Paul  Revere  aircraft,  one  of  the  pro¬ 
viders  of  near-real-time  MTI  at  Empire  Challenge; 

B:  interior  of  the  Paul  Revere  aircraft.  C,  D,  and  E: 
opposing  force  players  mimic  the  activities  of  insur¬ 
gents,  including  the  planting  and  detonation  of  impro¬ 
vised  explosive  devices;  F:  the  coalition  forces’ 
command  post,  with  the  FET  shown  on  the  projected 
display,  upper  left;  G:  detail  view  of  the  FET  display 
from  image  F;  H:  the  Tactical  Operations  Center  for 
Empire  Challenge,  at  China  Lake,  Calif. 
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Empire  Challenge 

The  NGA,  in  partnership  with  the  Joint  Forces  Command 
(JFCOM),  has  been  running  a  series  of  field  experiments 
called  Empire  Challenge.  For  the  past  few  years,  Empire 
Challenge  has  been  taking  place  at  the  China  Lake  Naval 
Weapons  Station  in  Ridgecrest,  Calif.,  in  the  northeast 
corner  of  the  Mojave  Desert.  With  a  focus  on  ISR  sensors, 
systems,  and  integration,  and  participation  by  coalition 
partners  (Britain,  Canada,  and  Australia),  it  has  become 
the  premier  annual  intelligence-related  field  exercise. 

Lincoln  Laboratory  has  been  participating  in  Empire 
Challenge  for  several  years,  flying  the  Paul  Revere  aircraft 
with  LiMIT  radar  and  instrumenting  ground  vehicles 
with  GPS  units  in  order  to  collect  ground  truth.  Start¬ 
ing  with  Empire  Challenge  2007  (EC07),  the  FET  began 
participating  in  the  exercise  (Figure  11). 

At  EC07,  the  FET  had  a  limited  presence  that  marked 
the  debut  of  the  FET  outside  of  Lincoln  Laboratory;  it 
was  essentially  a  proof  of  concept.  This  initial  release  of 
the  FET  performed  well.  Moreover,  the  design  choice  of 
going  with  a  network-centric  architecture  (requiring  only 
an  Internet  browser  and  the  Google  Earth  client  to  run 
the  FET)  paid  off  handsomely:  several  organizations  at 
EC07  discovered  the  FET  on  the  network  and  used  it  as 
an  additional  situation-awareness  display.  Two  particular 
examples  of  users  of  this  initial  FET  release  at  EC07  are 
the  NGA  GeoScout  Concept  Car  project  and  the  Canadian 
All-Source  Intelligence  Center. 

Participation  in  EC07  made  it  clear  that  for  FET  to 
be  a  serious  contender  in  a  tactical  system,  it  would  have 
to  be  modified  with  an  emphasis  on  scalability  and  real¬ 
time  operation.  In  particular,  it  would  need  to  be  able  to 
keep  up  with  a  live  stream  of  MTI  data  from  the  JSTARS 
aircraft.  The  other  lesson  was  that  full-motion  video  from 
a  ScanEagle  UAV  was  the  dominant  form  of  intelligence, 
and  an  effort  would  need  to  be  made  to  better  integrate 
ScanEagle  with  the  FET. 

At  Empire  Challenge  2008,  which  concluded  on 
August  1,  FET  was  a  huge  success.  In  particular,  our  recent 
work  on  scalability  yielded  positive  results:  on  the  final 
day  of  the  exercise,  the  FET  consumed  and  exploited— in 
real  time— 591,641  MTI  detections  from  JSTARS. 

The  FET  was  run  by  numerous  users  in  several  rooms 
at  China  Lake.  In  addition,  it  was  run  locally  on  Hanscom 
Air  Force  Base  at  the  Electronic  Systems  Center  Com¬ 
mand  and  Control  Enterprise  Integration  Facility,  with  a 


direct  connection  to  the  China  Lake  network,  where  the 
FET  server  was  located.  FET  data  were  also  sent  out  over 
SIPRNet  to  a  remote  site  as  well  as  to  Lincoln  Laboratory, 
allowing  us  to  provide  support  from  a  variety  of  locations. 

The  FET  was  used  to  directly  support  the  commander 
of  U.S.  forces  (Task  Force  Lancer)  as  part  of  the  Joint  Bat- 
tlespace  Awareness  ISR  Integration  Capability  effort.  FET 
provided  situation  awareness  that  was  available  through 
no  other  means.  It  also  routinely  protected  U.S.  ground 
forces  and  provided  tips  and  cues  to  the  ScanEagle, 
including  several  examples  of  direct  machine-to-machine 
tasking.  The  Task  Force  Lancer  commander  stated  that 
the  FET  added  “tremendous  situation  awareness”  to  his 
tactical  view  of  the  battlefield. 

With  the  positive  feedback  we  have  received  from 
our  participation  in  Empire  Challenge  2008,  our  main 
goal  for  the  upcoming  year  is  to  identify  an  operational 
site  or  tactical  unit  willing  to  serve  as  an  alpha  site  to 
explore  the  use  of  the  FET  within  an  operational  ISR 
community.  In  addition  to  continued  refinement  of  the 
exploitation  algorithms  and  work  on  performance  char¬ 
acterization,  we  are  also  hoping  to  add  the  ability  to  dis¬ 
play  and  exploit  signals  intelligence  (SIGINT).  At  EC08, 
SIGINT  reports  often  gave  key  clues  regarding  (simu¬ 
lated)  insurgent  activity.  SIGINT  would  be  a  complemen¬ 
tary  data  type  to  both  MTI  and  AIS,  and  would  allow  a 
variety  of  multi-intelligence  exploitation  algorithms  to 
be  investigated. 
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